What if I would tell you that Language Models and Multi-Agent Reinforcement learning are now engaged and will get married soon? First and foremost, kudos to Andrés Fernández Rodríguez who sent me the inspiring paper "Multi-Agent Reinforcement Learning is a Sequence Modeling Problem". The idea of the paper is fantastic. In its essence, it is about mapping the problem of agent control to token translation. The authors use an encoder-decoder model like the original "Attention is all you need" paper.

For Generative Deep Learning, having a high quality and high quantity is key. On the symbolic music files, this boils down to gathering as many MIDI files as possible. Fortunately, quite a few curated and annotated datasets are already available. Although pale in comparison to any Large Language Model dataset in NLP, there are some datasets that are quite extensive. After implementing my MIDI to dataset preprocessing engine a while ago, I keep myself busy preprocessing different music selections and then training GPT on them.

Not only did Transformer make their way successfully into Computer Vision just a short while ago, but they also contribute to the field of Neural Networks that work on different kinds of data. "PolyViT: Co-training Vision Transformers on Images, Videos and Audio" showcases a transformer that works on images, videos and audio. The idea behind transformers is to consider your input data as some form of a sequence of tokens. In NLP those tokens are discrete and usually mapped to the continuous plane of existence using embedding layers. Images on the other hand are typically cut into non-overlapping patches, which are then projected by some neural network layers to continuous vectors.